In digital communications, and in particular in connection-based communication systems such as asynchronous transfer mode (ATM) networks, a number of computers are interconnected to a network and communicate messages or streams of information in sequences of units, called packets or frames (used interchangeably herein). In asynchronous transfer mode networks, packets are a combination of small units called cells. A packet sent to a particular computer is received by the computer through a network interface which connects the computer to the network. Similarly, a computer injects packets into the network via the network interface. The task of the network interface is to process packets efficiently for both reception and transmission.
In this application, the term "message" signifies the semantic content of a packet, i.e., what the packet data represents.
In most current computer systems, a packet is received by capturing the packet first in a buffer in the network interface. At an appropriate time, the host computer is notified of the arrival of the packet. With some network interfaces, the host computer then copies the packet from the network interface into a free buffer in its host memory and performs any processing that is required by the packet. More recent network interfaces have a direct memory access (DMA) mechanism which allows the network interface to place the data into a free buffer in the host memory. Similarly, when a message is transmitted by the host computer, it informs the network interface of the location of the data in the host memory and the desired destination. The network interface copies appropriate size chunks of the data from the host memory into local memory of the network interface and then forms packets with that data for transmission to the destination. More recent network interfaces allow for direct memory access to the host memory to obtain data for outgoing packets.
The process of receiving and transmitting messages comprised of packets involves computer processing overhead which is chiefly due to operating system interaction and message copying. Operating system interaction typically occurs because application programs on the host computer do not interact directly with the network interface, but rather indirectly via the host operating system. There are four major types of interactions. First is handling notification, typically via an interrupt when packets arrive and/or when the number of unprocessed packets in the receive queue exceeds a given threshold. Second is application initiated transfer of data either to a transmit queue, to send a packet, or from the receive queue in response to notification from the network interface. Third is the translation of virtual memory addresses used by the application program into physical addresses used by the network interface to access host memory directly. In a state of the art system, on the transmit side this interaction occurs when the operating system informs the network interface of the data to be sent. On the receive side, this same overhead occurs when the operating system informs the network interface of the location, of a free buffer in which a received message should be placed. Fourth is general housekeeping functions, such as keeping the free buffer queue full, that are not in a critical path. Operating system calls and handling interrupts are typically expensive. Such operating system interaction both consumes host processing cycles, reducing the number of cycles available to application programs, and adds a delay in sending and receiving messages.
Message copying overhead generally occurs because the buffer(s) used by the operating system in which a received message is stored, is chosen from a global pool of available free buffers in the operating system, such that there is no fixed assignment of free buffers to connections. As a result, the location of the free buffer is not guaranteed to be the final location in the host memory in which the message data is desired by the application program. Therefore, the operating system copies the data from the free buffer to the desired final location. Thus, message copying also consumes host processing cycles, reducing the number of cycles available to application programs, and adds a delay in receiving messages.
Operating system interaction is reduced by using a network interface that permits an application program to access directly the network interface. However, to ensure appropriate protection, there is a mechanism which translates virtual addresses used by application programs to physical addresses used by the network interface. Such a network interface is disclosed, for example, in "Experiences with a High-Speed Network Adaptor: A Software Perspective," by Peter Druschel et al., published in the Proceedings of ACM SigComm '94, September 1994. This paper describes a system in which virtual addresses are used in network interface which includes a mapping engine to generate physical addresses from the virtual addresses. In such a system, the operating system does not need to perform the translation, but provides the network interface with a virtual address.
Message copying overhead can be reduced by providing a mechanism which transfers data directly from the network interface to the application program address space, bypassing the host processor. Such a system is disclosed in U.S. patent application Ser. No. 08/226,541 by Randy B. Osborne, filed Apr. 12, 1994, now abandoned. In such a system, messages communicated in the network include an area of destination control information, which identifies an operation to be performed by the network interface and possibly some operands to be used in performing the operation. A mechanism in the network interface examines a received message and performs the operation indicated by the destination control information without interaction with the host processor. For example, data in a message may be deposited directly into host memory at a location determined by the network interface to eliminate copying by the host computer.
Direct deposit of message data in host memory is just one kind of message processing that can be performed by a network interface using such control information. There are several other kinds of operations which involve low level processing of messages between retrieving a message from the host and injecting it in the network and receiving a message from the network and storing it to the host.
One kind of message processing is message filtering. With an appropriate message format, the receive side can filter out undesired messages before such messages arrive at the host and consume host resources, particularly host processor time. One possible use for such filtering is to increase the efficiency of ATM Forum-style LAN Emulation (LANE) implementations. Such implementations broadcast messages to find unknown destinations and echo messages sent to a LANE server. Both of these operations result in hosts receiving unwanted messages. Usually, a host computer consumes resources to process such messages only to immediately discard them. With the present invention, a message processor can inspect the control information in the beginning of a received message and discard the message without ever involving the host computer.
Another kind of message processing is fast and cheap demultiplexing of messages. The beginning of a message could be used as a demultiplexing field to indicate where a message should be sent in the host. For example, a transmission control protocol (TCP) port address can be extracted from the header of a TCP packet and used to send the packet to a queue appropriate for that port. This demultiplexing by the network interface and message coprocessor can help off-load message demultiplexing from the host.
Message processing also includes remote operations, which belong to a class of protocols in which messages carry information describing the action to take with the message body at the destination. In traditional protocols, the host computer at the destination cannot do any processing of a message until it receives all of the data it requires. As described above, this requirement introduces extra overhead of copying the data, and more significantly for small messages, notification overhead and interruption of the host processor main line activity for message processing. Some industrial control systems already use remote operations for implementing "cyclic" memory, known more generally as reflective memory. Message processing support in a network interface in accordance with the present invention makes implementation of cyclic memory easier and more efficient.
Another kind of message processing operation is scatter direct memory access (DMA). Two types of scatter DMA are particularly useful: implicit scatter, where the scattering sizes are predefined, and explicit scatter, where the message contains scatter destination address bundled with the message data. A example of implicit scatter is separating the protocol header from message data and sending the header to one buffer and the data to another buffer, where the buffers can be in arbitrary locations. This technique can be used in TCP/IP processing. For explicit scatter DMA, low level processing at the destination can unbundle the data and addresses and send the data to the appropriate addresses.
Providing support in a network interface for such message processing, however, adds complexity and therefore cost to the network interface. Since low level message processing in network interfaces currently involves research in non-standard communication protocols, this added cost generally outweighs the benefits in commercial applications. Accordingly, most commercially-available network interfaces do not provide such support.